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EXECUTIVE  SUMMARY 


A  large  percentage  of  Airway  Facilities  (AF)  maintenance  technicians  will  reach  retirement  age 
in  the  next  5  years.  As  the  number  of  technicians  decreases,  increased  demands  are  plactd  on 
the  remaining  workers.  Each  technician  i  o  longer  specializes  on  a  single  piece  of  equipment, 
hut  is  responsible  for  maintaining  several  different  pieces  of  equipment.  In  addition,  AF 
equipment  has  become  more  complex  with  advances  in  technology.  Technicians  must  work  hard 
to  maintain  proficiency  on  older  systems  while  learning  new  systems.  Advanced  technology 
training  systems  may  help  to  prepare  large  numbers  of  new  technicians  for  these  challenges. 

The  term  "advanced  technology”  refers  to  the  use  of  simulation.  Expert  Systems  and  Intelligent 
Tutoring  Systems  (ITS)  features  in  the  construction  of  a  computer-based  training  system.  Via 
interviews  and  site  visits  at  the  Federal  Aviation  Administration  (FAA)  Academy  and  regional 
training  centers,  it  was  found  that  currently  available  AF  training  does  not  take  advantage  of 
simulation,  Expert  System,  oi  ITS  technology. 

This  paper  describes  a  i  emonstration  training  system  that  was  deve’oped  to  investigate  the 
application  of  advanced  technology  to  AF  maintenance  training.  The  demonstration  training 
system  was  developed  with  the  assistance  of  current  and  retired  AF  technicians  for  the  Air 
Traffic  Control  Beacon  Interrogator  -  Series  4  (ATCBI-4).  The  ATCB1-4  training  system  allows 
technicians  to  practice  troubleshooting  on  simulated  malfunctions  with  the  help  of  an  expert 
advisor.  This  system  lepresents  the  latest  extension  to  the  Microcomputer  Intelligence  for 
Technical  Training  (MITT)  software  and  runs  on  Office  Automation  Technical  Support  (OATS) 
machines  in  the  Windows  environment. 

Preliminary  evaluation  of  the  system  indicates  that  advanced  technology  training  systems  can  be 
effective  for  AF  maintenance  training.  Futu:e  plans  include  the  development  at  an  authoring 
system  that  can  be  used  by  AF  technicians  ard  training  specialists  to  build  similar  (raining 
systems  for  other  types  of  AF  equipment  and  the  development  of  an  integrated  information 
system  which  combines  the  training  system  with  an  expert  job  aid  and  an  information  retrieval 
system. 


L  INTRODUCTION. 


A  large  percentage  of  Airway  Facilities  (AF)  maintenance  technicians  will  reach  retirement  age 
in  the  next  5  years.  As  the  number  of  technicians  decreases,  increased  demands  are  placed  on 
the  remaining  workers.  Each  technician  is  no  longer  able  to  specialize  on  a  single  piece  of 
equipment,  but  is  responsible  for  maintaining  several  different  pieces  of  equipment.  In  addition, 
AF  equipment  itself  has  become  more  complex  with  advances  in  technology,  further  increasing 
the  technician’s  cognitive  workload. 

Given  these  conditions,  AF  technicians  are  being  challenged  to  maintain  proficiency  on  older 
systems  while  learning  new  systems.  Training  on  new  systems  often  comes  several  months 
before  the  equipment  is  actually  in  place  and  functioning.  By  the  time  a  technician  is  asked  to 
apply  classroom  knowledge,  much  is  forgotten.  Advanced  technology  training  systems  may  help 
rectify  this  problem  by  providing  on-site,  individualized,  refresher/proficiency  training. 

2.  BACKGROUND. 

This  report  summarizes  the  work  completed  in  the  first  phase  of  a  three-phase  research  project 
on  the  application  of  advanced  technology  to  AF  maintenance  diagnostic  training.  This  section 
describes  the  three  phases  of  work  and  defines  what  is  meant  by  advanced  technology. 

2.1  THREE  PHASES  OF  RESEARCH. 

Table  1  lists  the  time  schedule  and  major  deliverables  for  each  phase  of  the  planned  research. 
All  three  phases  rely  on  the  cooperation  of  technicians  and  other  Federal  Aviation  Administration 
(FAA)  personnel  from  the  Southern  Region  and  the  FAA  Academy. 

The  first  phase  of  the  work  involved  two  major  activities.  First,  the  status  of  AF  proficiency 
training  was  investigated  to  determine  if  advanced  technology  was  currently  being  used.  The 
results  of  this  investigation  showed  that  improvements  could  be  made.  Thus,  the  second  activity 
produced  a  prototype  of  an  advanced  technology  training  system  for  AF  maintenance.  Sections 
3  and  4  of  this  report  discuss  the  two  Phase  I  activities  in  greater  detail. 

In  the  second  phase  of  this  research,  the  prototype  system  will  be  evaluated  for  student  and 
instructor  acceptability.  The  feedback  from  these  evaluations  will  be  used  to  convert  the 
prototype  training  system  into  a  complete  Intelligent  Tutoring  System  (ITS).  ITS’s  are  described 
in  section  2.2. 

During  Phase  II,  work  will  also  begin  on  an  AF  Training  Development  System  for  authoring 
additional  AF  Maintenance  ITS’s.  The  AF  Training  Development  System  will  allow  the  FAA 
to  capitalize  on  the  capabilities  of  advanced  technology  training  by  providing  tools  which 
facilitate  the  development  of  such  ITS  systems.  The  tool  will  also  facilitate  updating  of  the 
knowledge  in  such  training  systems.  The  first  step  will  be  to  identify  specifically  who  within 
the  AF  community  will  use  the  Training  Development  System.  Based  on  the  needs  of  target 


users,  a  functional  specification  of  the  AF  Training  Development  System  will  be  developed,  as 
well  as  a  demonstration  Development  System. 

In  the  final  activity  of  Phase  II,  the  research  team  will  produce  a  development  plan  and 
preliminary  functional  specification  for  an  advanced  technology  AF  Information  System.  The 
AF  Information  System  will  integrate  training,  real-time  job  aiding,  and  information  storage  and 
retrieval  into  one  system.  The  specification  will  detail  the  functions  of  the  AF  Information 
System,  based  on  an  analysis  of  technicians’  information  needs.  One  major  aspect  of  the 
development  plan  will  explain  how  the  expert  knowledge  base  from  the  training  system  "'ill  be 
extended/adapted  for  use  by  a  consultation  model  job-aid.  The  AF  Information  System  would 
provide  AF  Maintenance  technicians  with  a  tool  that  not  only  supports  maintenance  proficiency, 
but  also  provides  easy  access  to  documentation  and  expert  assistance  while  solving  problems  in 
real-time. 

The  third  phase  of  the  project  will  focus  on  the  design  and  development  of  first  the  Training 
Development  System  and  then  the  AF  Information  System.  The  research  team  will  work  with 
ar.  ACD-350  advisory  group,  the  Technical  Requirements  and  Evaluation  Team  (TRET),  to 
ensure  that  both  systems  are  aligned  with  FAA  expectations.  Once  the  Development  System  is 
complete,  two  workshops  will  be  held  to  instruct  FAA  personnel  how  to  use  the  Development 
System. 


TABLE  1.  PHASES  OF  RESEARCH  PLAN 


Phase 

Time  Schedule 

Major  Deliverables 

I 

August  1991  -  May  1992 

Technology  Status  Report 

AF  Maintenance  ITS  Prototype 

II 

May  1992  -  February  1993 

Complete  AF  Maintenance  ITS 
Specification/Demo  ITS  Development  System 

Plan  for  AF  Information  System 

III 

March  1993  -  March  1994 

Complete  ITS  Development  System 

AF  Information  System 

2.2  DISCUSSION  OF  ADVANCED  TECHNOLOGY. 

In  its  broadest  sense,  advanced  technology  refers  to  all  receni  innovations  in  hardware  (HW)  and 
software  (SW)  technology.  Only  a  small  subset  of  such  technology  is  applicable  to  technical 
troubleshooting  training.  In  particular,  this  research  focussed  on  advanced  technology  extensions 
of  traditional  computer-based  instruction.  Figure  1  shows  the  generic  model  for  traditional 
computer  based  instruction  (CBI).  As  the  figure  illustrates,  a  student  interacts  with  an 
instructional  environment  on  a  computer  .hrough  some  type  of  interface  (typically,  a  monitor  and 
keyboard). 


2 


± 


JL 


Jt 


In  early  CBI  systems,  the  information  presented  by  the  computer  was  limited  to  monochrome 
text  and  simple  line  graphics.  Today,  there  is  a  large  amount  of  new  (affordable)  HW  and  SW 
technology  that  permits  the  capture,  creation,  display,  storage,  and  retrieval  of  high  resolution 
color  graphics,  animations,  text,  and  video  (see  table  2).  Such  technology  allows  technicians 
to  interact  with  high  fidelity  images  of  equipment  and  permit  the  storage  of  extensive  amounts 
of  information. 


Computer-Based  instruction 

Student 

J 

L 

\ 

r 

Interface 

Keyboards,  touch  panels. 

i 

L 

graphics,  video,  audio, ... 

1 

r 

f 

Instructional  Environment 

Tutorials,  problems,  drill  Sc 
practice,  simulations, ... 

FIGURE  1.  GENERIC  CBI  ARCHITECTURE 


TABLE  2.  EXAMPLES  OF  ADVANCED  HW  AND  SW 


Capture 

Creation 

Display 

Storage/Retrieval 

Scanners 

Word  Processing  SW 

Graphics  Adapters 

Compact  Disc  ROM 

•  Hand-held 

•  VGA 

Laser  Disc 

flat-bed 

Graphics  SW 

640X480X16 

Interactive  Video  Disc 

•  Black  &  white 

•  Paint  packages 

320X200X256 

M-Motion  Video 

color 

•  Object-oriented 

•  Super  VGA 
640x480X256 

Digital  Video  Interactive 
PC- VCR 

Optical  Character 
Recognition 

Still  Video  Camera 

Cam-corders 

Animation  SW 

•  2-Dimensional 

•  3-Dimensional 

1280X1024 

Bar-code  Readers 

•  •  • 
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Advances  in  interface  technology  have  lead  to  the  use  of  graphical  user  interfaces  (GUI)  that 
allow  direct  manipulation  of  the  objects  on  the  screen  through  the  use  of  a  mouse,  touch  screen, 
track  ball,  joy  stick,  or  other  input  devices.  Such  direct  manipulation  devices,  in  conjunction 
with  an  interactive  simulation,  permit  technicians  to  "leam-by-doing." 

Instructional  systems  may  also  take  advantage  of  the  hypertext  and  hypermedia  SW  that  has  been 
developed.  Hypertext  SW  permits  developers  to  link  text  stored  in  a  computer  with  any  other 
text  and  to  provide  a  means  for  readers  to  access  these  non-linear  paths  (Horn,  1989). 
Hypermedia  applications  extend  this  concept  by  including  such  components  as  video,  graphics, 
animation,  sound,  and  photographs.  In  a  hypermedia  instructional  environment,  students  are  not 
restricted  to  a  single  prespecified  learning  sequence,  but  rather  may  use  links  (provided  by  the 
instructional  developer)  to  rapidly  browse  the  instructional  content  and  to  choose  an 
individualized  learning  path  through  the  content  (Nielsen,  1990). 

Advances  in  SW  technology  have  also  addressed  the  issues  related  to  making  computer-based 
training  more  "intelligent."  The  work  of  cognitive  scientists  and  artificial  intelligence 
researchers  over  the  past  decades  have  focussed  on  such  SW  technology,  including  simulations, 
Expert  Systems,  and  ITS’s.  Simulations  model  the  functionality  and  behavior  of  equipment  or 
systems  and  can  be  used  as  part  of  the  instructional  environment  of  a  training  system. 
Computer-based  simulations  have  been  shown  to  be  an  effective  tool  for  diagnostic  training 
(Johnson,  1981,  1987;  Maddox,  Johnson,  and  Frey,  1986).  Expert  Systems  model  the  problem 
solving  knowledge  of  a  human  expert  in  some  specific  area  (Hayes-Roth,  1987).  Often  this 
expert  knowledge  is  represented  in  the  form  of  if/then  rules;  however,  other  forms  of  knowledge 
representation  may  be  used  (e.g.,  frames,  scripts,  cases,  neural  networks).  Intelligent  Tutoring 
Systems  (Kearsley,  1987;  Poison,  and  Richardson,  1988;  Psotka,  Massey,  and  Mutter,  1988; 
Sleeman  and  Brown,  1982;  Wenger,  1987)  extend  the  basic  CBI  structure  by  adding  three 
models:  an  expert  model,  a  student  model,  and  an  instructor  model.  Figure  2  illustrates  the 
generic  structure  of  an  ITS. 

This  illustration  represents  the  conceptual  relationship  between  the  student,  the  instructional 
environment,  the  instructor  model,  the  student  model,  and  the  expert  model.  The  expert  model 
in  an  ITS  is  implemented  as  some  type  of  Expert  System.  Conceptually,  the  expert  model 
contains  the  domain  specific  knowledge  that  the  student  is  to  learn  (e.g.,  rules  for 
troubleshooting  malfunctions  in  a  piece  of  equipment).  The  student  model  is  a  dynamic 
hypothesis  of  what  the  faining  system  thinks  that  the  student  currently  knows  (e.g.,  a  list  of 
parts  which  may  be  causing  the  current  malfunction).  This  model  is  based  on  the  observable 
actions  that  the  student  performs  in  the  instructional  environment  and  on  the  expert  model  (e.g., 
performed  output  test  of  X,  viewed  error  indicators  on  control  panel  Y,  etc.).  The  instructor 
model  typically  compares  the  model  of  the  student  to  the  model  of  the  expert  in  order  to  provide 
feedback  to  the  student  (e.g.,  That  test  was  unnecessary  since...).  The  instructor  model  may 
also  monitor  the  student  model  to  decide  when  remediation  is  needed  and  to  select  future  lessons 
or  problems  (e.g.,  if  student  solved  N  problems  of  type  A  without  error,  give  a  problem  of  type 
B). 


Student 
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fnte/face 

Instructor 

tnstiuctfonat  Environment 

Expert 

Model 

i 

Model 

^  T  ^ 
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Compares  student 

Model 

Encoded  knowledge: 

to  expert  Structures 

rules,  connectivity,  etc. 

feedback,  remediation. 

Dynamic  account  of 

and  instruction. 

student  knowledge 

FIGURE  2.  GENERIC  ITS  ARCHITECTURE 


3.  STATUS  OF  ADVANCED  TECHNOLOGY  IN  AF  MAINTENANCE  TRAINING. 

Via  unstructured  interviews  and  site  visits  at  the  FAA  Academy  AF  Training  Development  Unit 
and  a  regional  CBI  training  center,  Galaxy  Scientific  investigated  the  current  application  of 
advanced  expert-system  technology  to  AF  maintenance  training.  Our  informal  investigation 
found  that  currently  available  AF  training  does  not  take  advantage  of  simulation,  Expert  System 
or  ITS  technology.  Rather,  the  majority  of  the  current  computer-delivered  training  is  based  on 
the  older  and  more  limited  CBI  architecture.  The  remainder  of  this  section  reports  the  current 
status  of  CBI  in  AF  maintenance  training  in  more  detail. 

3.1  OVERVIEW  OF  CBI  WITHIN  THE  AF  TRAINING  CURRICULUM. 

CBI  is  one  of  seven  types  of  training  delivered  to  AF  personnel.  Table  3  shows  the  distribution 
of  AF  courses  across  the  seven  types  of  training:  resident  training,  correspondence  study,  out- 
of-agency  training,  CBI,  on-the-job  training,  and  field-conducted  training.  Whereas  almost  50 
percent  of  all  AF  courses  are  delivered  through  resident  training  (at  the  FAA  Academy  in 
Oklahoma  City),  only  about  11.5  percent  of  all  AF  courses  are  delivered  through  CBI  (at  the 
regional  training  centers).  However,  the  total  number  of  resident  students  is  typically  only  about 
twice  the  total  number  of  CBI  students.  Thus,  as  one  would  expect,  the  relatively  small  number 
of  CBI  courses  serve  a  comparatively  high  volume  of  students. 
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TABLE  3.  AF  TRAINING  PROGRAM  CURRICULUM 


Type  of  Training 

Current  No. 
of  Courses 

Resident  Training 

231 

Computer-Based  Instruction 

35 

Correspondence  Study 

80 

On-The-Job  Training 

28 

Out-Of-Agency  Training 

70 

Field-Conducted  Training 

22 

Total 

466 

Of  the  41  AF  CBI  courses,  23  are  operational  courses,  9  are  demo  courses,  and  the  other  9  are 
refresher  courses.  Operational  courses  may  include  a  considerable  amount  of  lab  work  in 
addition  to  CBI.  Operational  courses  are  required  for  certification  and  include  a  final  test  which 
is  scored  and  recorded  on  the  trainee’s  permanent  record.  Demo  and  refresher  courses,  on  the 
other  hand,  are  not  required  for  certification.  In  fact,  a  student  does  not  receive  credit  for 
completing  a  demo  course;  however,  demo  courses  are  often  used  as  self-study  by  personnel 
who  want  to  sit  for  a  bypass  test.  Refresher  courses  serve  a  different  purpose  altogether. 
Experienced  personnel  take  refresher  courses  to  review  certain  material.  Such  refresher  courses 
include  practice  tests  and  upon  successful  completion  of  the  entire  course,  the  student  may  print 
a  certificate  of  completion. 

It  is  important  to  recognize  that  CBI,  therefore,  is  only  one  small,  albeit  important,  part  of  the 
overall  AF  training  curriculum.  There  appears  to  be  a  conscientious  effort  to  use  a  variety  of 
trairing  delivery  methods  to  serve  different  training  needs.  The  application  of  advanced 
technology  to  maintenance  training  could  allow  a  new  set  of  courses  to  be  delivered  at  technician 
sites  to  meet  the  need  for  refresher/proficiency  training. 

3.2  APPLICATION  OF  CBI  TO  AF  MAINTENANCE  TROUBLESHOOTING  TRAINING. 

The  application  of  CBI  to  AF  Maintenance  troubleshooting  training  has  been  somewhat  limited. 
Only  one  AF  CBI  course  is  directed  specifically  toward  diagnostic  training.  Course  47001 
(Troubleshooting  Principles)  is  an  operational  course  that  addresses  general  diagnostic  procedures 
rather  than  those  specific  to  a  particular  piece  of  equipment  or  system.  In  addition,  most  CBI 
courses  which  provide  training  for  specific  pieces  of  equipment  typically  include  one  or  more 
lessons  on  diagnosis. 

There  is  a  gap,  however,  in  the  area  of  refresher/proficiency  training.  After  initial  training  is 
received  on  a  piece  of  equipment  (typically  through  resident  training),  a  technician  returns  to  the 


field.  Ideally,  technicians  immediately  get  reinforcement  of  the  classroom  knowledge  by 
applying  the  classroom  knowledge  on-the-job;  however,  this  is  not  always  the  case.  Whenever 
there  are  lags  between  initial  learning  and  application  of  knowledge  or  v/hen  there  are  long 
periods  between  opportunities  to  practice  skills,  attrition  of  knowledge  and  skills  occurs.  Few 
refresher  courses  for  troubleshooting  are  currently  available;  consequently,  technicians  have  little 
opportunity  other  than  on-the-job  training  to  maintain  proficiency. 

3.3  EFFECTIVENESS  OF  AF  CBI. 

In  order  to  assess  the  AF  CBI  courses  that  are  currently  available,  two  visits  were  made  to  a 
regional  CBI  training  center.  A  Proficiency  Development  Specialist  (PDS)  at  the  center 
provided  assistance  by  giving  a  list  of  courses,  logging  onto  the  courses,  and  answering 
questions  as  the  content  of  the  courses  were  reviewed.  In  addition,  one  visit  was  made  to  the 
AF  Development  Unit  at  the  FAA  Academy  to  discuss  current  and  future  AF  CBI  with  a  small 
group  of  CBI  programmers  and  developers.  This  section  first  summarizes  the  status  of  the 
present  CBI  courses.  The  next  section  discusses  the  future  plans  for  AF  CBI. 

The  delivery  of  nearly  all  AF  CBI  is  currently  administered  through  the  Plato  system  at  regional 
training  centers.  The  average  age  of  the  Plato  lessons  is  greater  than  6  years  old.  The  general 
consensus  among  those  interviewed  (including  students,  instructors,  and  AF  training  developers 
and  programmers)  was  that  the  Plato  system  is  inadequate  for  a  number  of  reasons. 

Reason  1:  Slow  network.  First,  the  Plato  system  resides  on  a  mainframe  (CYBER)  at 
Oklahoma  City  and  it  is  accessed  from  remote  sites  with  1200  baud  modems.  Consequently, 
the  courses  delivered  through  this  system  have  very  slow  response  times  and  are  subject  to 
unavailability  whenever  some  part  of  the  communications  network  fails. 

Reason  2:  Unsophisticated  courses.  A  second  problem  with  the  Plato  system  is  the  nature  of 
the  courses  delivered  by  the  system.  Many  of  the  courses  are  simple  "page-turners"  composed 
primarily  of  text.  A  student  is  constrained  to  proceed  in  a  predetermined  sequence  through  the 
material.  Users  reportedly  find  this  type  of  instruction  uninteresting,  presumably  due  to  the  lack 
of  engaging  interaction.  In  most  cases,  this  type  of  CBI  training  would  be  more  economical 
(and,  given  the  slowness  of  the  network,  more  efficient)  in  book  form.  In  some  cases,  there  is 
an  even  greater  problem  than  the  simplistic  course  architecture.  During  their  review  of  current 
AF  CBI  courses,  the  AF  Training  Development  Unit  found  that  the  content  of  some  courses  is 
poorly  organized.  Such  courses  are  confusing  and  frustrating  for  students. 

Reason  3:  High  overhead.  A  third  problem  with  the  Plato  system  is  the  extensive  use  of 
"introductory"  screens.  Before  reaching  any  course  content,  a  student  must  often  go  through 
several  screens  containing  such  things  as  logos,  directions  on  using  the  interface,  and  objectives. 
The  slow  response  time  of  the  network  makes  it  a  time  consuming  process  to  go  through  all  of 
these  screens.  To  compound  this  problem,  several  courses  use  other  media  such  as  text  and 
video.  It  can  be  frustrating  to  go  through  so  many  screens  only  to  be  told  that  the  content  for 
the  course  is  located  off-line. 


Reason  4:  Unsupported,  linear  exercises.  A  fourth  problem  with  the  Plato  system  is  the 
sophistication  of  the  practice  exercises.  Some  exercises  are  multiple  choice  questions  which 
were  provided  as  practice  for  the  graded  test.  No  instruction  or  individualization  was  given 
within  the  linearly  ordered  questions.  If  the  student  answered  correctly,  positive  feedback  was 
given  and  the  next  question  was  presented.  If  the  student  answered  the  question  incorrectly, 
negative  feedback  was  given  and  the  same  question  was  presented.  No  hint  or  advice  was 
available  to  the  student. 

Some  of  the  newer  theory-based  courses  use  a  slightly  more  sophisticated  approach.  The  student 
is  pretested  on  the  material  for  each  lesson.  If  he/she  has  already  mastered  the  material,  the 
student  will  pass  the  pretest  and  may  then  skip  to  the  next  pretest.  This  allows  students  to  move 
more  quickly  to  the  material  they  don’t  already  know.  It  was  pointed  out  that  the  final  test  for 
the  course  is  required  for  all  students,  even  if  all  pretests  are  passed. 

Reason  5:  Limited  graphics.  A  fifth  problem  with  the  Plato  system  is  its  limited  graphics 
capabilities.  The  existing  system  is  capable  of  handling  only  monochrome  line  graphics.  The 
test  imbedded  in  such  graphics  is  often  quite  small  and  difficult  to  read.  Consequently,  the 
existing  system  is  unable  to  present  the  graphic  displays  necessary  to  convey  important 
information  for  diagnostic  training  and  to  hold  student  interest  (Johnson,  1987). 

3.4  AN  OATS  SOLUTION?. 

Due  to  frustrating  experiences  with  learning  on  the  Plato  system,  students  reportedly  have  a  very 
negative  attitude  about  CBI.  In  an  attempt  to  correct  the  problems  with  the  Plato  system,  a  new 
delivery  platform  has  been  adopted  called  Office  Automation  Training  System  (OATS).  The 
OATS  platform  was  selected  so  that  computer-delivered  instruction  can  be  delivered  on  a  stand¬ 
alone  system.  That  is,  the  courses  will  be  delivered  from  the  local  OATS  computer  without 
being  tied  to  a  remote  system  through  a  network.  The  course  material  will  be  stored  on  CD- 
ROM  and  updated  CD’s  will  be  distributed  as  needed.  For  security  reasons,  however,  testing 
will  still  be  handled  over  a  network. 

The  AF  Training  Development  Unit  has  already  received  positive  feedback  on  the  stand-alone 
course  concept.  The  research  team  agrees  that  freeing  students  from  the  slow  response  time  of 
the  network  is  an  important  step  to  improving  FAA  computer-based  training.  In  addition,  the 
AF  developers  are  planning  to  augment  the  basic  OATS  system  with  special  1IW  and  SW  to 
support  multi-media  learning  including  audio,  video  disk,  imaging,  graphics,  and  animation 
capabilities. 

Transitioning  to  the  new  platform  requires  translation  of  existing  courses  so  that  they  run  on  the 
new  machines.  Rather  than  "porting"  all  courses  to  the  new  platform,  the  AF  Training 
Development  Unit  first  reviewed  all  existing  CBI  courses  for  instructional  acceptability.  Those 
found  to  be  unacceptable  are  being  discontinued,  while  the  courses  which  have  acceptable 
content  are  being  upgraded  during  the  conversion  to  the  stand-alone  format.  The  upgrading 
involves  the  use  of  the  multi-media  capabilities  of  the  OATS  platform  as  well  as  a  break  from 


the  simple  page-turner  format.  The  AF  Training  Development  Unit  at  the  FAA  Academy  also 
indicated  their  plans  to  make  use  of  Expert  Systems  technology  for  individualized  instruction, 
refresher/proficiency  training,  and  resident  courses  where  appropriate. 

Unfortunately,  the  development  of  courseware  for  the  OATS  platform  is  not  proceeding  as 
planned  due  to  problems  with  the  authoring  language.  In  the  past,  Tutor,  TenCore,  and  Quest 
were  used  to  develop  courses  for  the  Plato  system.  With  the  move  to  the  OATS  platform,  a  new 
authoring  language  called  Accord  has  been  adopted.  There  are  several  problems  with  the 
Accord  language  which  are  interfering  with  the  development  of  updated  and  new  courseware. 
The  developers  indicated  that  there  is  a  longer  learning  curve  for  getting  proficient  in  using  the 
Accord  language  compared  to  TenCore.  Reasons  given  included  that  there  is  limited 
documentation  available  for  Accord  and  some  of  the  documented  syntax  did  not  work  as  stated. 
The  other  complaint  was  that  the  resulting  programs  have  slow  transitions  between  screens 
(approximately  4-5  seconds).  As  a  result,  the  FAA  is  now  considering  a  new  authoring  system 
(e.g,  Authorware  Professional  or  Asymetrix  Toolbook). 

In  summary,  the  current  AF  CBI  is  not  addressing  the  AF  troubleshooting  training  needs. 
Although  AF  Training  personnel  have  begun  implementing  plans  to  improve  the  current  state 
of  computer  delivered  instruction,  there  is  room  for  the  addition  of  an  advanced  technology 
training  system.  Specifically,  there  is  a  need  for  an  ITS  which  provides  a  supported  practice 
environment  for  learning  how  to  troubleshoot  specific  AF  equipment. 

4.  PROTOTYPE  ADVANCED  TECHNOLOGY  TRAINING  SYSTEM. 

The  prototype  advanced  technology  training  system  was  developed  over  a  6-month  period. 
Sections  4.1  discusses  the  selection  of  the  instructional  domain  and  section  4.2  describes  the 
development  and  delivery  HW  and  SW.  The  actual  development  process  involved  iterations 
between  knowledge  acquisition,  design,  implementation  and  evaluation.  Section  4.3  focusses 
on  the  knowledge  acquisition  component  of  the  development  process.  Section  4.4  then  describes 
the  components  of  the  final  prototype  training  system. 

4.1  SELECTION  OF  THE  INSTRUCTIONAL  DOMAIN. 

The  first  step  in  developing  a  prototype  of  an  advanced  technology  training  system  is  to  identify 
a  piece  of  AF  equipment  that  is  appropriate  for  this  type  of  training  and  for  which  additional 
training  is  needed.  The  goal  of  this  research  effort  was  to  develop  a  prototype  training  system 
that  could  be  used  immediately  by  AF  technicians  in  the  field. 

The  initial  candidates  for  the  instructional  domain  suggested  by  FAA  personnel  included  the 
Paradyne  modem,  Air  Traffic  Control  Beacon  Interrogator-4  (ATCBI-4)  and  ATCBI-5,  and 
Common  Digitizer.  These  pieces  of  AF  equipment  were  suggested  because  current  training  was 
either  unavailable  or  insufficient.  After  holding  discussions  with  FAA  personnel  during  site 
visits  to  the  General  National  Airspace  Sector  (GNAS)  office  in  Atlanta,  GA,  and  the  Air  Route 
Traffic  Control  Center  (ARTCC)  in  Hampton,  GA,  the  ATCBI-4  was  selected  as  best  meeting 


the  criteria  for  selection  shown  in  table  4.  The  Paradyne  Modem  was  rejected  for  the  prototype 
development  due  to  the  lack  of  available  troubleshooting  expertise.  The  Common  Digitizer  was 
rejected  because  the  training  need  was  not  as  great  as  the  ATCBI-4.  The  ATCBI-4  and  ATCBI- 
5  are  very  similar  pieces  of  equipment.  The  ATCBI-4  was  selected  over  the  ATCBI-5  because 
of  logistics:  the  research  team  was  located  in  close  proximity  to  an  ATCBI-4  system  and  expert 
technicians. 


TABLE  4.  SELECTION  CRITERIA  FOR  INSTRUCTIONAL  DOMAIN 


System  Characteristics 

Technician  Needs  and  Constraints 

•  It  is  a  complex  technical  system. 

•  It  is  not  too  complex  for  rapid 
development  of  a  prototype. 

•  It  and  its  test  equipment  have  a 
variety  of  instruments. 

•  It  has  multiple  test  points. 

•  It  requires  a  small  but  critical 
amount  of  troubleshooting. 

•  They  need  practice  to  be 
proficient. 

•  They  cannot  practice  on  the  real 
equipment. 

•  They  need  procedural  information 
to  diagnose  the  system. 

4.2  DELIVERY  AND  DEVELOPMENT  HW  AND  SW. 

In  order  for  the  prototype  training  system  to  be  of  value  to  the  technicians  in  the  field,  it  had 
to  be  developed  for  an  HW  delivery  system  that  would  be  available  to  the  technicians.  Given 
that  the  FA  A  had  selected  OATS  machines  as  the  new  standard,  the  prototype  advanced 
technology  training  system  was  targeted  for  delivery  on  an  OATS  machine  in  the  Microsoft 
Windows  environment.  The  prototype  requires  the  following  minimal  configuration:  386/25 
megahertz  (MHz)  machine,  Microsoft  Windows  3.0,  4  megabytes  (Mb)  of  memory,  video 
graphics  array  (VGA)  monitor,  a  run-time  version  of  Asymetrix  Toolbook,  and  a  mouse  input 
device. 

The  initial  prototype  development,  however,  began  in  the  DOS  environment  in  order  to  make 
use  of  rapid  prototyping  tools:  Microcomputer  Intelligence  for  Technical  Training  (MITT)  and 
MITT  Writer.  Both  MITT  and  MITT  Writer  run  in  the  DOS  environment  and  use  EGA 
graphics.  Figure  3  illustrates  the  relationship  between  these  two  tools.  MITT  Writer 
(Wiederholt,  1991)  is  a  development  environment  that  allows  users  to  produce  a  description  of 
training  without  the  need  to  write  computer  code.  MITT  (Johnson,  Norton,  Duncan,  and  Hunt, 
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FIGURE  3.  RELATIONSHIP  BETWEEN  MITT  AND  MITT  WRITER 
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1988;  Norton,  Wiederholt,  and  Johnson,  1991)  uses  the  training  description  Files  produced  by 
MITT  Writer  to  deliver  simulation-oriented  troubleshooting  training. 

In  addition  to  its  rapid  prototyping  capability,  the  MITT  system  is  an  ITS  which  provides  two 
types  of  advice:  procedural  and  functional.  MITT’s  functional  advice  is  based  on  the  simulation 
model  which  represents  the  connections  between  parts.  MITT’s  procedural  advice  is  based  on 
documented  troubleshooting  procedures  and  the  experience  of  expert  technicians. 

Although  MITT  Writer  was  designed  to  aid  an  instructor  or  domain  expert  in  developing  an 
MITT  tutor,  this  tool  was  used  by  a  member  of  the  research  team  to  quickly  produce  and  modify 
an  initial  prototype  system.  The  MITT  program  was  then  used  to  run  the  training  system  to 
obtain  input  and  feedback  from  technicians.  Technicians  who  reviewed  the  initial  prototype 
system  reacted  favorably  to  the  MITT  approach  to  troubleshooting  training.  Thus,  the  research 
team  decided  to  capitalize  on  the  MITT  technology  to  develop  a  new  simulation-based  ITS  that 
would  run  in  the  Windows  environment.  Asymetrix  Toolbook  and  Borland  C++  were  used  to 
develop  the  Final  prototype. 

Figure  4  illustrates  the  architecture  of  the  Final  prototype  training  system.  The  ATCBI-4 
Training  Program  contains  the  main  training  system  interface  and  content.  This  program  then 
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FIGURE  4.  ARCHITECTURE  OF  PROTOTYPE  TRAINING  SYSTEM 


accesses  four  other  specialist  programs  as  needed:  the  MITT  system,  the  part  output  value 
database,  the  on-line  manual,  and  the  advice  system. 

4,3  DESCRIPTION  OF  THE  KNOWLEDGE  ACQUISITION  PROCESS. 

The  process  used  to  develop  the  prototype  advanced  technology  training  system  involved 
iteration  between  knowledge  acquisition,  SW  development,  and  evaluation.  The  research  team 
worked  closely  with  technicians  at  the  Radar  Unit  located  at  GNAS  in  Atlanta,  GA,  throughout 
the  development  phase. 

Initially,  a  single  technician  was  identified  to  serve  as  the  ATCBI-4  Subject  Matter  Expert 
(SME).  One  member  of  the  research  team  worked  closely  with  this  technician  to  gain  an 
understanding  of  the  ATCBI-4  system  and  the  technician’s  troubleshooting  task.  The  initial 
knowledge  acquisition  efforts  also  relied  heavily  on  existing  training  materials  and  the  ATCBI-4 
manufacturer’s  manual.  The  SME,  however,  is  the  critical  resource  in  the  knowledge 
acquisition  process.  This  became  particularly  apparent  when  the  initial  SME  was  sent  to  the 
FAA  Academy  for  a  6-week  training  course.  Other  technicians  were  involved  in  the  project  at 
this  point;  however,  similar  conflicts  continued  to  occur.  The  problem  was  solved  by  hiring  a 
recently  retired  ATCBI-4  technician  who  could  devote  his  full  attention  to  the  knowledge 
acquisition  process. 

In  the  remainder  of  this  section,  an  example  of  a  critical  knowledge  acquisition  task  will  be  used 
to  illustrate  the  knowledge  acquisition  process.  The  example  task  is  the  development  of  a 
functional  flow  diagram.  A  functional  flow  diagram  illustrates  the  parts  of  a  system  and  the 
functional  connections  between  parts.  Such  diagrams  are  similar  to  functional  block  diagrams 
and  schematics  that  are  common  in  technical  systems.  Therefore,  the  researcher  and  an 


experienced  ATCBI-4  technician  used  existing  block  diagrams  and  schematics  as  a  starting  point 
for  this  task.  Several  iterations  were  needed  to  develop  the  final  functional  flow  diagram  used 
in  the  prototype. 

A  key  choice  in  the  development  of  a  functional  flow  diagram  is  the  level  of  detail  that  is 
represented.  For  example,  since  the  ATCBI-4  system  is  primarily  an  electronic  system,  the 
parts  could  be  fundamental  electronics  components  (e.g.,  resistors,  capacitors)  or  entire 
subsystems  (e.g.,  receiver,  transmitter).  In  general,  the  answer  depends  on  the  level  at  which 
troubleshooting  takes  place.  For  most  malfunctions  in  the  ATCBI-4  system,  technicians  replace 
parts  at  the  card  level.  However,  the  technicians  indicated  that  they  do  not  solve  the 
troubleshooting  task  by  thinking  about  flow  of  signals  between  cards.  Rather,  technicians 
analyze  symptoms  and  make  tests  based  on  more  abstract  functional  entities  which  may  include 
several  fundamental  components  located  on  different  cards.  In  order  to  support  the 
troubleshooting  process,  the  parts  in  the  ATCBI-4  functional  flow  diagram  represent  this 
intermediate  block  level. 

During  this  process,  MITT  Writer  was  used  to  construct  and  modify  the  SW  representation  of 
the  parts  and  their  connections.  The  resulting  SW  representation  of  connections  serves  as  the 
foundation  of  the  ATCBI-4  simulation  and  the  knowledge  base  for  the  functional  expert. 
Therefore,  as  changes  were  made,  the  SME  was  given  printouts  of  the  database  (i.e. ,  a  listing 
of  parts  and  their  connections)  and  asked  to  check  the  accuracy  of  the  SW  representation  of  the 
functional  flow  diagram.  A  similar  process  of  interviewing,  developing,  and  reviewing  was 
followed  throughout  the  knowledge  acquisition  process. 

4,4  DESCRIPTION  OF  THE  PROTOTYPE. 

The  prototype  investigates  the  use  of  advanced  technology  in  proficiency  training  for  AF 
technicians  in  the  area  of  troubleshooting.  The  instructional  domain  for  the  prototype  (ATCBI-4) 
is  a  complex  electronics  system  which  is  represented  at  the  functional  block  level  rather  than  at 
the  component  level.  That  is,  each  functional  block  represented  in  the  training  system  is 
composed  of  several  lower  level  components  in  the  actual  system.  The  prototype  allows 
technicians  to  practice  troubleshooting  on  simulated  malfunctions  (at  the  functional  block  level) 
with  the  help  of  an  expert  advisor.  Since  there  is  no  single  "correct"  way  to  troubleshoot  this 
system,  the  advisor  is  primarily  passive.  However,  student  actions  are  tracked  to  give  the 
proper  advice  when  it  is  requested.  The  remainder  of  this  section  describes  the  prototype 
training  system  in  more  detail. 

4,4.1  The  Four  Sections  of  the  Training  System. 

The  prototype  training  system  is  divided  into  four  major  sections:  Introduction  to  Training, 
Understand  the  Simulation,  Understand  the  System,  and  Practice  Problems.  Figure  5  shows 
the  main  screen  of  the  training  system.  The  four  buttons  at  the  bottom  of  the  screen  allow  the 
user  to  access  any  of  the  four  sections.  The  Practice  Problems  section  is  the  primary  focus  of 


FIGURE  5.  MAIN  SCREEN  OF  PROTOTYPE  TRAINING  SYSTEM 
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the  proficiency  training  system,  while  the  first  three  sections  provide  background  and  support 
information.  Descriptions  of  each  of  the  four  sections  follow. 


4.4. 1.1  Introduction  to  Training  Section. 


The  Introduction  to  Training  section  orients  the  new  user  to  the  overall  ’raining  system.  It 
includes  a  brief  explanation  of  the  four  different  sections  and  provides  exercises  for  users  who 
are  unfamiliar  with  mouse-driven  SW.  These  exercises  allow  the  novice  user  to  learn  how  to 
interact  with  various  direct  manipulation  objects  such  as  buttons,  menus,  and  dialog  boxes.  For 
example,  figure  6  shows  the  screen  that  instructs  the  user  how  to  interact  with  buttons.  This 
section  also  points  out  the  various  help  features  that  are  avadable  in  the  system. 
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Help  Map  Exit 


INTERACTING  WIT^M  BUTTONS  •/•  '  *  \  ^Pagq-7,  •! 


Exritef 


Amplifier 


1.  TO  SCROLL  TEXT 

Clide  on  the  DOWN 
arrow  to  see  the  text 
below,  or 

Click  the  UP  arrow  ta 
see  the  text  above. 

. .  •  ..  Try  it. 


There  are  several  different  types  of  “buttons" 
that  you  will  use  in  this  training  system.  You 
may  have  already  used  the  buttons  at  the 
bottom  of  the  screen  to  go  to  different  screens. 
Some  examples  of  other  types  of  buttons  are 
shown  on  the  left.  Try  the  following  exercises: 

Exercise  t: 

Click  the  "Do  Test"  button. 

Click  the  "Exciter"  box. 

Click  on  the  "Continue"  button. 

Click  the  arrow  between  the  "Doubler11  and 
"Amplifier1' 

Exercise  2: 

Follow  the  directions  given  in  the  green  box. 

Exercise  3: 

Click  the  "ON"  text. 

Click  the  "OFF1  text. 


FIGURE  6.  EXAMPLE  INTRODUCTION  TO  TRAINING  SCREEN 


4.4. 1.2  Understand  the  Simulation  Section . 

The  Understand  the  Simulation  section  acquaints  the  new  user  with  the  simulation  that  he  will 
interact  with  in  the  Practice  Problems  section.  Before  attempting  a  practice  exercise,  it  is 
helpful  for  technicians  to  know-  what  information  is  available  to  them  and  how  to  access  this 
information. 

The  Understand  the  Simulation  section  states  such  fundamentals  as  the  purpose  of  the  simulation, 
the  assumptions  that  were  made  in  developing  the  simulation,  and  how  to  start  and  stop  the 
simulation.  The  technician  can  also  learn  about  the  two  types  of  tests  that  can  be  simulated 
during  practice  problems  (output  tests  and  single  part  tests)  and  the  different  methods  for 
performing  each  test.  In  addition,  this  section  explains  the  two  types  of  advice  that  are  available 
during  a  practice  problem  (functional  and  procedural  advice)  and  provides  interactive  examples 
of  each. 
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This  section  also  allows  the  technician  to  become  familiar  with  the  simulated  displays  of  the 
ATCBI-4  and  test  equipment  that  they  can  use  during  a  practice  exercise.  For  example,  figure 
7  shows  the  simulated  System  Monitor  Panel  of  the  ATCBI-4.  If  the  technician  is  unfamiliar 
with  a  particular  gauge  or  control,  he  can  click  on  that  control  to  get  a  description  of  it.  The 
technician  can  also  click  on  control  settings  to  obtain  notmal  meter  readings.  By  accessing  such 
display  screens  from  the  Understand  the  Simulation  section,  the  technician  can  learn  how  the  real 
system  has  been  mapped  to  the  simulation. 


4,4. 1,3  Understand  the  System  Section . 

The  Understand  the  System  section  allows  technicians  to  review  information  about  the  system 
equipment  that  is  being  simulated  and  to  become  acquainted  w'ith  the  reference  material  that  is 
available  on-line.  The  technician  can  read  descriptions  of  each  part  of  the  equipment  represented 
in  the  simulation  as  well  as  explanations  of  how  to  test  the  part  itself  and/or  its  output.  The 
technician  can  also  access  various  types  of  reference  material  such  as  tables  containing  standard 
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and  tolerance  values  as  well  as  preventive  procedures  that  are  relevant  to  troubleshooting.  The 
prototype  contains  a  minimal  amount  of  such  on-line  documentation  to  illustrate  the  basic 
information  storage  and  retrieval  capability  that  will  be  implemented  in  a  full-scale  AF 
Information  System. 

In  addition,  the  technician  can  interact  with  the  schematic-like  functional  flow  diagrams.  These 
diagrams  show  how  the  parts  of  the  system  are  functionally  connected  to  one  another.  Figure 
8  shows  the  Transmitter  Flow  Diagram.  The  buttons  at  the  bottom  of  this  screen  allow  the 
technician  to  perform  tests  and  get  part  and  test  information  from  the  diagram.  For  example, 
to  perform  a  test  on  the  output  of  the  exciter,  the  technician  would  click  on  the  "Do  Test"  button 
then  click  the  line  that  connects  the  exciter  block  to  the  doubler  block.  A  box  appears  with  the 
results  of  the  test.  The  colors  of  the  blocks  and  lines  are  then  updated  to  reflect  the  status  of 
parts  and  outputs  that  are  known  as  a  result  of  the  test.  Gray  indicates  that  the  status  is  not 
known,  green  indicates  that  the  status  is  normal,  and  red  indicates  that  the  status  is  abnormal. 
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FIGURE  8.  EXAMPLE  FUNCTIONAL  FLOW  DIAGRAM  SCREEN 
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In  the  Understand  the  System  section,  all  parts  and  outputs  are  normal  initially  so  that  the 
technician  can  review  normal  system  values.  If  the  technician  wants  to  see  how  the  system 
would  change  if  a  certain  part  has  failed,  he  can  select  a  part  to  fail  in  a  "What  if..."  exercise. 
The  technician  selects  a  part  to  fail  by  using  the  functional  flow  diagram.  Once  a  selection  has 
been  made,  the  diagrams  are  updated  automatically  to  indicate  what  outputs  are  affected  by  the 
failed  part.  Parts  and  outputs  that  are  normal  appear  in  green,  while  those  that  are  abnormal 
are  colored  red. 

4. 4.1. 4  Practice  Problems  Section. 

The  Practice  Problems  section  allows  the  technician  to  practice  the  mental  skills  of 
troubleshooting  with  the  help  of  an  expert  advisor.  When  the  technician  starts  a  practice 
problem,  a  scenario  is  given  which  states  the  initial  conditions  of  a  problem.  Figure  9  shows 
an  example  of  a  problem  scenario.  The  technician  is  then  free  to  begin  troubleshooting.  The 
technician’s  goal  is  to  identify  the  malfunctioning  component  as  quickly  as  possible  without 
making  unnecessary  tests  and  without  replacing  parts  that  are  functioning  normally. 


'System  Monitor  pi  S 


During  the  daily  checks  you  found  no  power  out  at 
J28.  Begin  troubleshooting  to  find  out  what  part  is 
causing  this  problem. 


Start  a  problem  by  clicking  on 
the  "New  Prob"  button  below.  A 
JbjoxjwiiL-annear  stating  the  initial 
jthe  problem,.  Some 
wilt  also  appear, 
jmbleshooting.  • 

is  to  identify  the 
part  within  the  time 
will  be  informed  if 
ecessary  tests  or  if 
arts  that  are  normal. 

nulation  displays  by 
the  buttons  in  the 
the  left  Go  to  the 
ow  diagrams  by 
wuuwuy '  uir  -•the  buttons  below. 
Use  the  menus  at  the  top  of  the 
screen  at  any  time. 


FIGURE  9.  EXAMPLE  PRACTICE  PROBLEM  SCENARIO 
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After  reading  the  problem  statement,  the  technician  is  free  to  choose  how  he  wants  to  solve  the 
problem.  In  general,  the  technician  can  go  to  the  simulation  displays  or  functional  flow 
diagrams  and  begin  choosing  tests  to  perform.  There  are  two  levels  of  information  that  can  be 
accessed  as  the  result  of  requesting  a  test  to  be  performed.  Tests  performed  on  the  functional 
flow  diagrams  or  through  the  use  of  the  Tests  menu,  however,  always  state  the  summarized  test 
result,  i.e.,  whether  the  part  or  output  is  normal  or  abnormal.  By  reducing  the  complexity  of 
the  information,  the  technician  can  focus  on  learning  more  generalized  troubleshooting  skills. 

Technicians  who  want  a  higher  fidelity  simulation  may  use  the  simulation  displays  which  provide 
a  more  realistic  decision  environment.  On  the  simulation  displays,  the  test  results  are  given  as 
actual  data  values  which  must  be  interpreted  as  being  normal  or  abnormal.  For  example,  the 
oscilloscope  simulation  display  is  shown  in  figure  10.  The  technician  selects  a  test  point 
(consisting  of  a  card  number  and  pin  number)  and  the  waveform  is  displayed.  The  technician 
must  then  decide  if  this  is  normal  output  for  the  selected  test  point.  If  needed,  the  technician 
may  request  an  explanation  of  the  waveform  which  describes  the  waveform  and  states  whether 
it  is  normal  or  abnormal. 
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To  view  a  waveform, 
select  the  test  point  to  be 
viewed  by  placing  the 
cursor  on  the  item  in  the  list 
and  pressing  the  mouse 
button  once. 

The  waveform  will  appear 
in  the  top  left  corner  of  this 
screen. 

To  get  an  explanation  of 
the  waveform,  click  the 
"Explain*  button  below  the 
waveform. 


FIGURE  10.  EXAMPLE  OSCILLOSCOPE  TEST 
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If  the  technician  does  not  know  what  tests  to  perform,  a  request  can  be  made  for  either 
functional  or  procedural  advice.  The  functional  advisor  will  suggest  that  the  technician  perform 
a  test  on  the  part  that  has  the  potential  for  eliminating  the  most  parts  from  the  set  of  parts  that 
may  be  causing  the  malfunction.  Functional  advice  is  based  on  the  functional  flow  connections 
between  parts  contained  in  the  simulation  of  the  equipment  and  logical  troubleshooting 
principles.  For  example,  if  a  part  has  bad  output,  then  any  part  "downstream"  of  that  part 
cannot  be  causing  the  malfunction  and  can  be  eliminated  from  the  set  of  feasible  parts. 
Similarly,  if  a  part  has  normal  output,  then  any  part  "upstream"  from  that  part  cannot  be  the 
malfunctioning  part  and,  therefore,  can  be  eliminated  from  the  set  of  feasible  parts.  The 
functional  flow  diagrams  maintain  a  visual  record  of  the  feasible  set  by  updating  the  colors  of 
parts  and  outputs  as  tests  are  performed. 

The  procedural  advisor,  on  the  other  hand,  will  suggest  a  test  based  on  the  experience  of  an 
expert  technician.  A  recently  retired  AF  technician  was  consulted  to  develop  the  procedural 
advice  for  each  practice  problem.  The  expert  suggested  realistic  tests  for  quickly  isolating  the 
malfunctioning  part.  However,  it  must  be  emphasized  that  there  is  no  single  "correct"  procedure 
in  terms  of  which  part  to  test  in  what  order.  Therefore,  the  technician  is  not  forced  to  follow 
the  exact  steps  suggested  by  the  expert. 

There  are  times  when  the  technician  is  penalized,  however.  The  system  tracks  the  students 
actions  and  provides  unsolicited  instructional  advice  when  the  technician  does  one  of  the 
following:  (a)  tests  a  part  which  is  "upstream"  from  a  part  that  was  tested  and  shown  to  be 
normal,  (b)  tests  a  part  which  is  "downstream"  from  a  part  that  was  tested  and  shown  to  be 
abnormal,  and  (c)  identifies  a  normal  part  as  the  malfunctioning  part.  These  three  actions  count 
as  troubleshooting  errors  that  are  tallied  and  reported  at  the  end  of  a  practice  problem  session. 
In  addition,  the  system  keeps  track  of  how  many  times  the  technician  replaces  a  part  which  is 
normal.  If  this  action  occurs  more  than  a  preset  number  of  times,  then  the  system  informs  the 
technician  that  he  should  use  output  tests  to  isolate  the  fault  rather  than  replacing  parts  to  find 
the  one  that  is  malfunctioning. 

4.4.2  Description  of  the  ITS  Modules. 

Section  2.2  presented  the  generic  ITS  architecture  with  its  four  primary  components:  the 
Instructional  Environment,  the  Expert  Model,  the  Student  Model,  and  the  Instructional  Model. 
This  section  will  describe  these  four  components  as  implemented  in  the  prototype  training 
system. 

4.4.2. 1  The  Instructional  Environment. 

The  Instructional  Environment  is  the  basic  program  with  which  the.  student  interacts  to  learn. 
In  this  system,  a  simulation  is  used  by  the  student  to  practice  troubleshooting  the  ATCBI-4.  The 
main  question  in  the  implementation  of  the  simulation  is  the  determination  of  the  appropriate 
level  of  simulation  fidelity  for  the  learning  task. 


Simulation  fidelity  has  been  defined  by  Johnson  (1984)  along  two  dimensions:  physical  and 
psychological.  Physical  fidelity  refers  to  the  degree  to  which  the  simulator  looks,  feels,  and 
functions  like  the  real  equipment.  Psychological  fidelity  refers  to  the  degree  to  which  the  user 
of  the  simulation  must  call  upon  the  cognitive  processes  required  to  troubleshoot  the  real 
equipment.  The  ATCBI-4  training  system  prototype  provides  a  higher-degree  of  psychological 
fidelity  than  physical  fidelity  in  order  to  support  learning  of  the  cognitive  skills  of  fault  isolation 
in  troubleshooting. 

The  ATCBI-4  simulation  provides  adequate  but  limited  physical  fidelity.  The  look  of  the 
ATCBI-4  equipment  is  achieved  through  graphical  representations  of  the  system  displays  and  test 
equipment  in  the  simulator.  Such  graphics  do  not  provide  as  high  a  level  of  physical  fidelity  as 
a  full-HW  emulation  system,  however,  the  graphics  are  sufficient  to  attract  and  hold  student 
interest.  The  feel  of  the  ATCBI-4  equipment  is  provided  by  direct  manipulation  of  some  of 
controls  through  the  use  of  a  mouse.  The  simulation,  however,  does  not  support  all  of  the  low 
level  physical  skills  of  troubleshooting  (e.g. ,  setting  up  the  oscilloscope  to  make  a  test).  Finally, 
the  functionality  of  the  ATCBI-4  system  is  represented  by  a  part  simulation. 

The  part  simulation  of  the  ATCBI-4  models  the  block  level  components  of  the  equipment  and 
the  connections  between  these  parts.  Normal  and  abnormal  values  are  maintained  in  files 
indexed  to  a  particular  malfunction.  Given  the  malfunctioning  component,  the  simulation 
automatically  computes  which  parts  have  normal  and  which  have  abnormal  output.  Although 
this  level  of  simulation  was  satisfactory  for  the  prototype  system,  deeper  simulation  is  being 
considered  for  future  work.  Although  deeper  simulation  requires  more  work  initially  to  develop 
and  verify  the  models  which  transform  inputs  to  outputs,  this  effort  earns  savings  when  adding 
new  troubleshooting  problems. 

Although  physical  fidelity  is  important,  the  focus  in  this  work  was  on  developing  the 
psychological  fidelity  of  the  ATCBI-4  simulation.  Psychological  fidelity  is  achieved  by 
providing  the  information  resources  that  are  available  to  the  technician  during  fault  isolation  in 
troubleshooting.  This  information  begins  with  a  realistic  problem  scenario.  The  statement  of 
the  problems  should  map  io  realistic  situations  encountered  by  the  technician. 

Psychological  fidelity  is  also  important  during  fault  isolation.  Access  to  test  information  and 
reference  information  is  needed  to  support  the  cognitive  aspects  of  the  troubleshooting  process. 
Throughout  the  simulated  fault  isolation  process,  technicians  choose  what  tests  to  perform  and 
what  information  to  gather  as  they  would  in  the  real  system.  They  also  interpret  what  to  do  next 
based  on  the  results  of  each  test.  However,  since  this  is  a  learning  environment,  help  is 
available  to  the  technician  to  support  selection  of  tests  and  interpretation  of  test  results.  Advice 
about  what  tests  to  perform  and  feedback  for  student  actions  relies  on  the  expert,  student,  and 
instructional  modules. 
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4.4.2.2  The  Expert  Model. 


The  Expert  Model  in  the  ATCBI-4  encodes  two  types  of  expertise:  functional  and  procedural. 
Both  types  of  advice  were  discussed  from  a  user’s  point-of-view  in  section  4.4. 1.4.  This  section 
will  focus  on  the  implementation  of  the  two  types  of  advice. 

The  functional  expert  is  closely  coupled  to  the  simulation.  The  functional  expert  uses  the 
simulation’s  representation  of  part  connections  to  implement  a  half-split  type  algorithm.  Foi 
each  problem,  the  functional  expert  maintains  a  dynamic  list  of  feasible  parts  (i.e.,  parts  which 
may  be  causing  the  malfunction).  Initially,  all  parts  are  in  the  feasible  set.  Each  time  a  test  is 
made,  the  functional  expert  updates  the  list  of  feasible  parts,  based  on  the  functional  connections 
between  the  parts  and  the  status  of  the  part  for  the  current  malfunction.  For  example,  if  the 
output  of  part  A  is  tested  and  shown  to  be  normal,  then  the  functional  expert  would  eliminate 
all  parts  that  feed  part  A  from  the  list  of  feasible  parts,  since  these  parts  cannot  be 
malfunctioning.  When  a  technician  asks  for  functional  advice,  the  functional  advisor  applies  a 
half-split  strategy  to  the  feasible  set,  i.e.,  it  suggests  the  technician  test  the  output  of  the  part 
which  has  the  potential  for  eliminating  the  most  parts  from  the  feasible  set. 

The  functional  expert  is  essentially  a  Baysian  problem  solver,  achieving  maximum  information 
gain  per  action.  Because  the  functional  expert  is  context-free  (i.e.,  it  has  no  knowledge  or 
understanding  of  the  technical  domain),  it  always  has  an  answer,  based  only  on  the  functional 
connectivity  among  the  components.  Research  in  the  late  70’s  and  throughout  the  80’s 
demonstrated  that  the  functional/logical  approach  to  diagnostic  training  is  effective.  The 
approach  was  shown  to  be  effective  in  such  training  domains  as  aircraft  mechanics  (Johnson  and 
Rouse,  1982),  nuclear  power  plant  technicians  (Johnson,  Maddox,  and  Kiel,  1984),  U.S.  Army 
electronics  troubleshooting  (Johnson  and  Fath,  1984),  NASA  space  shuttle  diagnosis  (Johnson, 
et  al.,  1988),  and  in  other  domains. 

The  procedural  expert  is  a  rule-based  Expert  System.  The  procedural  models  the  knowledge  of 
expert  troubleshooters  in  the  form  of  if-then  type  rules.  The  "if"  part  of  each  rule  contains 
conditions  that  must  be  met  for  the  advice  to  apply,  such  as  "If  part  J  is  in  the  feasible  set  and 
the  output  of  part  K  has  not  been  tested."  The  "then"  part  of  each  rule  contains  the  advice  that 
is  given  and  explains  what  test  to  make  next.  In  the  prototype  system,  a  procedure  is  encoded 
for  each  problem.  When  a  technician  requests  procedural  advice,  the  procedural  expert  looks 
for  a  match  in  the  condition  of  each  rule  of  the  procedure  in  reverse  order,  so  that  the  most 
appropriate  test  is  suggested. 

4. 4. 2.3  The  Student  Model. 

The  Student  Model  of  the  ATCBI-4  prototype  tracks  student  actions  in  the  simulation.  The 
student  actions  that  are  tracked  include:  tests  performed  (single  part  and  output)  and  requests 
for  advice  (functional  and  procedural).  The  student  model  also  maintains  a  count  of  the  student 
errors  that  are  detected  by  the  instructional  model  (described  in  the  next  section).  At  the  end 
of  a  problem,  the  student  model  is  summarized  with  the  following  information:  how  many  times 
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all  parts  that  feed  part  A  from  the  list  of  feasible  parts,  since  these  parts  cannot  be  > 

malfunctioning.  When  a  technician  asks  for  functional  advice,  the  functional  advisor  applies  a 

half-split  strategy  to  the  feasible  set,  i.e.,  it  suggests  the  technician  test  the  output  of  the  part 

which  has  the  potential  for  eliminating  the  most  parts  from  the  feasible  set. 

The  functional  expert  is  essentially  a  Baysian  problem  solver,  achieving  maximum  information  i 

gain  per  action.  Because  the  functional  expert  is  context-free  (i.e.,  it  has  no  knowledge  or 
understanding  of  the  technical  domain),  it  always  has  an  answer,  based  only  on  the  functional 
connectivity  among  the  components.  Research  in  the  late  70’s  and  throughout  the  80’s 
demonstrated  that  the  functional/logical  approach  to  diagnostic  training  is  effective.  The 
approach  was  shown  to  be  effective  in  such  training  domains  as  aircrrft  mechanics  (Johnson  and  i 

Rouse,  1982),  nuclear  power  plant  technicians  (Johnson,  Maddox,  and  Kiel,  1984),  U.S.  Army 
electronics  troubleshooting  (Johnson  and  Fath,  1984),  NASA  space  shuttle  diagnosis  (Johnson, 
et  al.,  1988),  and  in  other  domains. 

The  procedural  expert  is  a  rule-based  Expert  System.  The  procedural  models  the  knowledge  of  ( 

expert  troubleshooters  in  the  form  of  if-then  type  rules.  The  "if"  part  of  each  rule  contains 

conditions  that  must  be  met  for  the  advice  to  apply,  such  as  "If  part  J  is  in  the  feasible  set  and 

the  output  of  part  K  has  not  been  tested."  The  "then"  part  of  each  rule  contains  the  advice  that 

is  given  and  explains  what  test  to  make  next.  In  the  prototype  system,  a  procedure  is  encoded 

for  each  problem.  When  a  technician  requests  procedural  advice,  the  procedural  expert  looks 

for  a  match  in  the  condition  of  each  rule  of  the  procedure  in  reverse  order,  so  that  the  most 

appropriate  test  is  suggested. 

4.4. 2. 3  The  Student  Model. 

The  Student  Model  of  the  ATCBI-4  prototype  tracks  student  actions  in  the  simulation.  The  ! 

student  actions  that  are  tracked  include:  tests  performed  (single  part  and  output)  and  requests 

for  advice  (functional  and  procedural).  The  student  model  also  maintains  a  count  of  the  student 

errors  that  are  detected  by  the  instructional  model  (described  in  the  next  section).  At  the  end 

of  a  problem,  the  student  model  is  summarized  with  the  following  information:  how  many  times 

i 
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advice  was  used,  how  many  errors  were  made,  how  many  tests  were  performed,  and  the  time 
it  took  for  the  student  to  solve  the  problem.  A  performance  index  is  also  given  based  on  this 
data. 

The  Student  Model  described  above  is  a  "within"  problem  model.  That  is,  the  model  tracks  the 
performance  of  a  student  within  a  single  problem.  As  more  problems  are  added  to  the  training 
system  during  Phase  II  of  this  research,  the  student  model  can  be  extended  to  include  an 
"overall"  student  model.  That  is,  a  model  that  records  the  types  of  problems  that  the  technician 
has  attempted  and  how  well  the  student  has  performed  on  each  type  of  problem.  Such  a  model 
could  then  be  used  to  support  intelligent  problem  selection,  rather  than  random  or  linear  problem 
selection. 

4. 4. 2. 4  The  Instructional  Model. 

The  Instructional  Model  in  the  ATCBI-4  prototype  detects  errors  in  student  actions  by  comparing 
student  actions  to  the  functional  expert  model.  Such  errors  are  based  on  a  few  general  rules 
for  correct  troubleshooting  behavior.  For  example,  if  a  part  has  been  eliminated  from  the  set 
of  feasible  parts  by  previous  student  actions,  then  it  is  an  error  for  the  technician  to  perform  a 
test  of  this  part.  Finally,  the  instructional  model  checks  for  unrelated  tests  of  outputs  or  parts. 
For  example,  if  the  technician  performs  a  test  of  a  part  whose  output  does  not  reach  any  part 
which  is  known  to  have  abnormal  output,  the  test  is  considered  an  error. 

It  should  be  noted  that  the  Instructional  Model  in  the  ATCBI-4  prototype  focuses  on  logical 
rather  than  procedural  troubleshooting  errors.  The  instructional  model  does  not  compare  the 
technicians  actions  to  the  steps  of  the  procedural  expert  because  there  is  no  single  "correct"  way 
to  troubleshoot  the  system.  However,  the  instructional  model  can  be  extended  in  Phase  II  to 
reinforce  the  expert  procedures  by  providing  the  step  by  step  procedure  for  review  by  the 
technician  at  the  completion  of  each  problem. 

In  addition,  the  instructional  model  can  be  extended  to  include  a  taxonomy  of  the  types  of 
problems  that  are  encountered  in  troubleshooting.  Such  a  taxonomy  provides  the  curriculum  that 
is  to  be  taught  by  the  training  system.  Furthermore,  individual  problems  can  be  rated  in  terms 
of  difficulty.  As  alluded  to  in  the  section  on  student  modeling,  such  indexing  can  be  used  as 
the  basis  for  selecting  the  most  appropriate  problem  for  the  current  student. 

4.5  DISCUSSION  OF  THE  PRELIMINARY  EVALUATION. 

An  informal  preliminary  evaluation  was  conducted  with  the  prototype  SW.  Three  technicians 
were  observed  while  interacting  with  the  prototype  training  system  during  approximately  1-hour 
sessions.  None  of  the  technicians  had  used  the  prototype  prior  to  this  session. 

Each  session  began  with  the  first  screen  of  the  training  system  already  displayed.  The 
developers  informed  the  technician  that  the  purpose  of  the  session  was  to  interact  with  the 
training  system  to  determine  what  they  like  and  dislike  about  the  system.  Each  technician  was 
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also  informed  that  there  would  be  an  evaluation  sheet  provided  at  the  end  of  the  session  to  solicit 
such  feedback.  A  copy  of  the  evaluation  sheet  is  included  in  the  appendix. 

In  general,  the  three  technicians  were  able  to  use  the  system  with  little  input  from  the  observers. 
No  major  errors  were  encountered;  however,  minor  changes  to  the  interface  were  made  in 
response  to  user  comments.  The  technicians  all  gave  positive  verbal  comments  about  the  system 
and  each  was  able  to  successfully  complete  a  practice  troubleshooting  problem.  Two  of  the 
three  technicians  indicated  that  they  would  use  this  system  for  proficiency  training.  (The 
technician 

who  indicated  that  he  would  not  use  this  system  is  a  supervisor  and  is  no  longer  responsible  for 
troubleshooting  the  ATCBI-4  equipment.) 

5.  PLANS  FOR  PHASE  II. 

In  the  second  phase  of  this  research,  the  prototype  system  will  be  extended  into  a  complete 
proficiency  training  system  for  troubleshooting  the  ATCBI-4.  There  are  four  major  tasks 
involved  in  this  effort.  First,  input  will  be  obtained  from  the  AF  Development  Unit  and  AF 
technicians  to  guide  development  of  the  complete  training  system.  Based  on  this  input,  changes 
will  be  made  to  extend  the  Instructional  Environment,  Student  Model,  Instructor  Model,  and/or 
the  Expert  Model.  Third,  knowledge  engineering  work  will  continue  in  order  to  extend  the  set 
of  simulated  practice  problems  to  include  approximately  10  to  12  malfunctions.  Finally,  the 
knowledge  base  of  domain  specific  data  will  be  restructured  to  improve  the  access  time.  The 
completed  ATCBI-4  training  system  will  be  installed  at  the  GNAS  radar  unit  in  Atlanta,  GA, 
and  will  be  made  available  for  distribution  to  other  FAA  radar  sites. 

Upon  completion  and  review  of  the  final  ATCBI-4  training  system,  work  will  begin  on  a 
development  system  for  authoring  additional  AF  Maintenance  ITS’s.  The  research  team  will 
work  with  personnel  at  the  FAA  Academy  to  develop  a  functional  specification  to  meet  the  needs 
and  desires  of  training  development  personnel.  A  demonstration  development  system  will  be 
developed  to  provide  a  concrete  basis  for  the  discussion. 

The  final  activity  of  Phase  II  will  be  the  development  of  a  plan  and  preliminary  specification  for 
an  advanced  technology  AF  Information  System.  The  AF  Information  System  will  integrate 
three  components  into  one  system:  training,  job-aiding,  and  information  storage  and  retrieval. 
The  training  system  completed  in  this  phase  will  serve  as  the  model  for  the  training  component 
of  the  AF  Information  System.  The  research  plan  will  detail  how  the  knowledge  base  of  the 
training  system  will  be  extended  to  support  real-time  job  aiding.  In  addition,  the  plan  will 
describe  the  development  plan  for  the  development  of  the  on-line  documentation  system. 

6.  CONCLUSIONS. 

This  research  has:  (a)  investigated  the  current  status  of  Airway  Facilities  (AF)  training  and 
found  that  simulation,  Expert  System,  and  Intelligent  Tutoring  System  (ITS)  technology  has  not 
been  used  in  the  computer  based  training  that  is  currently  available  to  AF  technicians,  and  (b) 
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produced  a  prototype  advanced  technology  proficiency  training  system  that  uses  simulation, 
Expert  System,  and  ITS  software  (SW)  technology  and  which  is  delivered  on  affordable 
hardware  (HW).  The  work  in  future  phases  will  seek  to:  (a)  develop  an  authoring  system  to 
permit  non-programs  to  build  similar  types  of  training  systems,  and  (b)  to  extend  the  prototype 
system  into  an  AF  Information  System,  integrating  the  training  system  with  real-time  job  aiding 
and  information  retrieval. 
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APPENDIX  A 

EVALUATION  FORM  FOR  ATCBI-4  PROTOTYPE 

Rate  the  following  on  a  scale  of  1  to  5  where  1  is  the  lowest  rating  and  5  is  the  highest  rating. 
Your  Background: 

_ 1.  Familiarity  with  the  ATCBI-4  system. 

(1  =  have  not  worked  on  it  in  past  2  years  +  5  =  work  with  it  daily) 

_ 2.  Experience  using  mouse-driven  software. 

(1  =  no  experience  5  =  extensive  experience) 

ATCBI-4  Prototype  Tutor 

_ 1.  The  usefulness  of  the  training  system  for  refresher/proficiency  training. 

(i  =  not  very  useful  5=  very  useful) 

_ 2.  The  "user  friendliness"  of  the  system. 

(1  =  frustrating  to  use  5  =  easy  to  use) 

_ 3.  The  accuracy  of  the  simulation  information. 

(i  =  not  very  accurate  5  =  very  accu’.ate) 

_ 4.  The  likelihood  you  would  use  this  training  system  if  it  were  available. 

(1  ~  never  5  =  frequently) 

COMMENTS 

In  the  space  provided  below,  indicate  any  additional  comments  you  have  about  the  four  sections 
of  the  training  system,  including  suggestions  for  improvement. 

Comments  about  Introduction  to  Training 


» 


i 


> 


Comments  about  Understand  the  Simulation 


\ 


Comments  about  Understand  the  System 


Comments  about  Practice  Problems 


Miscellaneous  Comments  (on  back) 
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